Data Maturity Assessment Impact Report 2022
Libby Harkins
27/05/2022
Executive Summary
The results of Data Orchard’s 2022 Impact Analysis of the Data Maturity Assessment tool were similar to the 2021 impact analysis. The tool is regularly used for leaning about data maturity. Taking an assessment often sparks discussion with colleagues which can lead to further action.
In this report we look at the results from this year’s impact analysis as well as comparing the results to last year’s findings and reflecting on what this means.
What we did
We surveyed 87 users of our Data Maturity Assessment tool, targeting those who had taken an assessment between November 2020 and March 2022. We asked about their experience of taking a Data Maturity Assessment and any benefits they experienced or actions they took as a result of taking an assessment.
Key Findings
People, commonly, take a data maturity assessment at least in part to learn more about data maturity. When they have taken a data maturity assessment they are likely to agree that they have learned about data maturity.
Organisational Data Maturity Assessments had more extensive benefits than individual Data Maturity Assessments,
Taking a data maturity assessment prompts sharing and discussion within organisations,
Applications for funding following a data maturity assessment were almost always successful,
40% of organisations go on to implement a data strategy fter taking an assessment,
The achievements of this strategy are not all that extensive. This may be because it is too soon to measure such achievements, or a weakness in our Theory of Change.
We do not have much data from service providers.
Introduction
Data maturity is the organisational journey towards improvement and increased capability in using data. Data Orchard has been researching data maturity since 2015 and have developed both a Data Maturity Framework and Data Maturity Assessment tool.
Our theory of change suggests that users experience key problems such as:
a lack of knowledge about data maturity (i.e. what good and great looks like);
a sense of needing to get better but not knowing where to start;
struggling to engage leaders and colleagues in discussions about data or driving improvements.
Our theory of change predicts that by taking a Data Maturity Assessment, the user experiences immediate benefits that bring about tangible actions. These actions lead to results that increase data maturity and provide long-term rewards and benefits. See the appendix for more detail about our data maturity framework and our theory of change.
In order to test our theory of change, we asked people who had taken the Data Maturity Assessment about their experience and the outcomes they witnessed.
Methods
Who was invited?
We sent out 707 invites to people who had taken our Data Maturity Assessment tool between the dates 6th November 2020 and 31st March 2022. 21 of these invites were to people who had led a paid-for organisational data maturity assessment with us. 686 were users of the free tool. In total, we received 70 complete responses and 18 partial responses.
| Individual | Organisational | Total | |
|---|---|---|---|
| Invites Sent | 686 | 21 | 707 |
| Complete Responses | 63 | 7 | 70 |
| Partial Responses | 16 | 1 | 17 |
| Total Responses | 79 | 8 | 87 |
| Response Rate | 11.5% | 38% | 12% |
The overall response rate is 12%. Response rate is higher for paid-for organisational users (38%) than free users (11.5%). And within free users, higher for those who used the full version (13%) than the 5 minute taster (8%). This response rate is considerably lower than for our last impact survey (which was 20% overall).
What did we ask them?
We used the same survey as our 2021 Data Maturity Assessment Impact Report. We used an online survey that asked respondents:
about the conditions in which they took the Data Maturity Assessment and their motivations for taking one
about their overall impression of the Data Maturity Assessment and benchmarking tool
any immediate benefits they received
any actions they took as a result of using the Data Maturity Assessment
We also asked if they had implemented a data strategy, and any achievements they gained from doing so, as well as if they had applied for or received funding.
Demographics of Respondents
We asked respondents in what capacity they took a Data Maturity Assessment. A proportion of respondents did not complete the survey beyond answering this question. The breakdown is shown in Figure 3.
Figure 4: Number of people completing impact survey by DMA user type
Results: This Year
Usefulness
63% of all respondents said they found the Data Maturity Assessment to be at least very useful and no one said they found the Data Maturity Assessment not at all useful. 59% of not-for-profit users found the Data Maturity Assessment very or extremely useful. Of the two respondents who found it not so useful:
one said that they needed more support to fill it out and lack of engagement from leaders led to no action
the other said that they found the questions irrelevant and hard to answer
Figure 5: How useful respondents found the data maturity assessment by user type
Benchmarking
The benchmarking feature held some importance to most users. It was mostly ‘somewhat important’, suggesting that it isn’t the driving reason for using the tool. This is supported by the fact that few respondents agreed that comparing their data maturity to others in the sector was a motivating factor in taking the assessment (see Motivation section).
Figure 6: Importance of the benchmarking feature by user type
Expectation vs Reality
Most users expected their organisation to be either average or worse than average (77%). People generally aren’t taking the assessment because they think they are doing better than others. We see this again when only 2% of not-for-profit free users cite ‘thought we might be doing quite well compared to others and wanted to check’ as a motivation for taking the assessment (see Motivation section). Most organisations compared to others in a way the user expected (63%).
Likelihood to Recommend
Of 69 people who answered this question, 68% are likely or very likely to recommend our assessment tool. Only 2% are unlikely.
Figure 7: How likely users of the data maturity assessment are to recommend it to others
We asked those who recommend the tool to give some details as to how they have benefitted and why they would recommend it. Some of the responses that have consented to being shared are:
“It’s accessible, comprehensive and viable for NfPs financially”
“It has given us an externally validated way of looking at our data maturity and the results have enabled us to start discussions on a data strategy. We have a good idea of where we need to go and where to prioritise as a result of carrying out the maturity assessment.”
“I’m sure my answers will be different in a year, so much has happened over the last two years, but now we have our data being collected in a coordinated and consistent way, I am expecting to see great things! We are already using data to influence our decision making process.”
“… We shared the results with our leadership team to benchmark transparently where we were at, build a plan to improve based on the results, and advocated for resources to implement our improvement plan. The results pushed our leadership to make progress on our data maturity. We set a target for increasing our data maturity from a 2.1 to a 3.0 in one year, using the assessment as a tool to measure progress, and were able to meet our goal.”
“This was a great tool for giving us a shared language to assess and discuss the issues and where we needed to improve. We are struggling to find the right resources to take it forward fully but have implemented key gaps - thank you.”
Those who were unlikely to recommend the tool expressed a difficulty in understanding where the numbers had come from.
Not-for-profit free-users analysis
Most users of the free tool completed the assessment individually (66%). 22% of users completed it as one of a number of individuals from the same organisation taking the assessment. 15% completed it in a group setting.
Respondents were asked to indicate the nature of their interest in data maturity by selecting from a list. The largest group were Data Champions (36%). 21% indicated ‘other’ and provided a range of responses including Data Strategists, an ICT coordinator, and managers of various kinds (Data & Performance, Evaluation & Impact, research leader).
Figure 8: Users by the nature of their interest in data in their organisation
Motivations
For users of our free tool, the most common motivation for taking the assessment was to learn more about data maturity as well as wanting to get better with data but not knowing where to start. Testing its suitability for colleagues to complete to help people in the organisation think/talk about data was also a popular answer. Other reasons included being asked to do it by a colleague and wanting to check how well the organisation is doing.
Figure 9: Motivations for taking the data maturity assessment for not-for-profit free users
Outcomes
Benefits
The most common benefit cited by users of the free tool was to gain an objective reflection of where the organisation is at (around 80% experience this moderately or extensively). Increased motivations and raised aspirations within the organisation were also frequently chosen answers, along with improved understanding about important factors and questions.
Figure 10: Benefits of taking a data maturity assessment for not-for-profit free users
Actions
The most common action was discussing changes with colleagues, followed by using the results to guide plans and sharing results with others in the organisation. 75% respondents chose at least one of these three options. Around 15% didn’t do any of the actions we proposed.
Figure 11: Actions taken by not-for-profit free users following the data maturity assessment
Five users (10%) sought internal funding/resources, all five were successful in securing this funding. These resources were generally used for training (60%), new tools (40%), new jobs (60%) and consultancy (40%). One sought external funding but was unsuccessful.
Impact / Achievements
Out of 52 who responded to our question about implementing a data strategy/improvement plan, 23 (44%) said they had done implemented a plan since completing the assessment. These plans resulted in the following achievements:
Figure 12: Impacts or achievements resulting from a data maturity assessment for not-for-profit free users
The top three achievements stated were increased knowledge and expertise, increased collaboration and sharing with partners/stakeholders and improved strategic planning and decision making.
Discussion/Summary
Overall, these results support our theory of change. Most users of the free tool were able to gain an objective reflection of where they are at and engaged in discussions with colleagues after taking the assessment. We see this as an important first step to improving data maturity.
The top motivation for taking an assessment was to learn more (19% of respondents chose this option). 70% of respondents agreed that improved understanding about important factors and questions was a benefit of taking the assessment. This suggests that people are gaining what they hoped from the tool: a greater understanding of data maturity.
10% didn’t take any of our proposed actions. Although it is promising to see that using our assessment tool leads to positive action for most people, it would be useful to know why for 10% of people it doesn’t. If these people are simply wanting to learn about data maturity, then then committing no further actions is not surprising. However, in order for our tool to have long term implications in the not-for-profit sector we need it to incite positive changes.
Although not a great number of users sought funding, it is of note that almost all of them were successful and therefore it is possible that the Data Maturity Assessment tool could be a useful aid in securing funds. Our theory of change predicts that these funds (which were spent on for training, new tools, new jobs) could lead to long term rewards. For example, increased credibility, influence, knowledge and expertise.
42% of respondents went on to implement a data strategy, although not many achievements were made ‘extensively’ from this. It could be that it is too soon to see long term achievements. Of the 63 not-for-profit free users who responded to our survey, 33 (52%) took the Data Maturity Assessment for the first time less than 6 months before. Additionally, it could be that there are other achievements of a data strategy that out theory of change does not address. Or that the strategies people are implementing are not that effective.
Service Providers Analysis
Service providers are organisations that support non-profits to assess and improve their data maturity. They may be consultants or infrastructure organisations. They self-select for this description.
Motivations
5 service providers responded to the question about their motivations for completing a Data Maturity Assessment.
2 were motivated by testing its suitability for the client to use and to help clients work out where to start improving
1 person was motivated by each of: learning more about data maturity, engaging senior leaders, demonstrating sector comparison, diagnosing how they’re doing and on completing the assessment on behalf of their organisation
None were motivated a by baseline to check impact of support
Outcomes
Benefits
Benefits for service providers were fairly broad, the two most significant benefits being raising awareness of what good practice looks like and reassurance that they were on the right track in their support or advice for the client. No respondents specified ‘other’ benefits they received.
Figure 13: Benefits experienced by service providers
Actions
3 service providers responded to our question about actions:
2 said the client invested in training and selected nothing else
1 said they secured consultancy, adapted new tools and created new roles
No one said created new jobs, or other.
Discussion/Summary
It is difficult to make any strong conclusions from these results as not many service providers completed the survey. This could be by chance, or because very few service providers user our data maturity assessment tool. For those who do use it, it seems as though the tool supports service providers by reassuring them and reinforcing what good practice looks like. Some qualitative data about how this fed into the support they were providing would help us to understand better.
Results: Compare to 2021
Not-for-profit free-users
Overall, the results from not-for profit users of the free tool are very similar to what we found for not-for-profit users last year.
Last year, a higher portion of respondents were motivated by wanting to learn more about data maturity, and considerably less were motivated by the inability to the unlock value of data for decision making.
Figure 14: Motivations for taking the assessment from respondents of the 2021 and 2022 impact survey
Benefits were similar again, with a higher proportion of respondents reporting moderate benefits in 2021 compared to 2022.
Figure 15: Benefits of taking the data maturity assessment from respondents of the 2021 and 2022 impact survey
It seems that more people reported taking actions after taking the assessment in 2021 compared to 2022. The trend of which actions were taken most is the same, except more people shared their results with the organisation in 2021.
Figure 16: Actions taken after a data maturity assessment from respondents of the 2021 and 2022 impact survey
Overall, The trend of results is similar for both the 2021 and 2022 data maturity assessment impact surveys. Respondents to the 2021 survey gave more notable results (i.e. stated more extensive benefits and conducted more actions following the assessment). This could indicate a decline in the effectiveness of our tool, or a sampling bias. As the data maturity assessment was less widely known in November 2020 when the last survey was sent out, the respondents may have been more invested in the tool and therefore will see more pronounced outcomes.
Service Providers
As there were only 3 service providers who responded to our survey this year, and 7 who responded last year, the sample size is too small to make any meaningful comparisons. We hope that over the next year we will be able to gather more data to investigate this.
Results: Organisational Users
We combined results from our last impact report to gain a greater sample size that allows us to investigate the impact of our paid for organisational assessment.
Across both Data Maturity Assessment impact surveys, 13 organisational users answered past the first question (5 last year, 7 this year).
Motivations
Similar to users of the free tool, organisational leads were motivated by wanting to learn more about data maturity and not knowing where to start. However, they weren’t as impacted by feeling as though they were doing worse than peers or struggling to engage leadership. Possibly because, in some cases, a data lead may have to secure a level of support in order to commission an organisational assessment.
People who chose ‘other’ mainly specified wanting a baseline or benchmark within their organisation and to check their progress.
Figure 17: Motivations for organisational leads to take a data maturity assessment
Outcomes
Benefits
Similar to free users, the top benefits were increased motivations and raised aspirations within the organisation. As well as gaining an objective reflection of where the organisation is at. Importantly, organisational assessments have experienced more extensive benefits than the free tool users. Although there are several reasons this could be, such as:
They are more invested in data maturity (taken the time to do organisation assessment)
Often organisational assessments are based within a plan to improve data
Figure 18: Benefits experienced by organisations an organisational data maturity assessment
Actions
10 of 12 organisational users (83%) who answered this question went on to discuss changes with colleagues. Many shared the results, used them to guide plans for a data strategy and secured leadership support. No-one took no action.
Figure 19: Actions taken by organisational leads following data maturity assessment
4 organisations sought funds to implement improvements and all were successful: This year, 3 sought funding/resources from internal budgets. Last year, 1 sought external funding. These funds were mainly spent on training (75%) and consultancy/external support (50%). New jobs, new roles and purchasing a management system were also mentioned.
Impact / Achievements
Of 12 respondents, 5 implemented a data strategy/plan after the assessment (41%). 2 from this year and 3 from last year. As a result of this plan, there were limited achievements. Strategic planning was achieved moderately by all.
Figure 20: Impact of data maturity assessment on organisations that had taken organisation data maturity assessment
Discussion/Summary
Many of the patterns we see in organisational users of the Data Maturity Assessment are similar to that of free users. The high motivation to learn about data maturity and the most common actions being about sharing results and discussing data maturity with colleagues. Importantly, most organisational users experienced all the benefits we asked about at least moderately. Furthermore, many more benefits were experiences ‘extensively’ to that of free users.
The fact that no respondents said they took no action after taking the assessment is support for the impact of our tool as a driver of change. Around 40% implemented a data strategy plan with limited achievements.
Appendix
Data Maturity
Data maturity is the journey an organisation takes to becoming better at using data. Our framework outlines seven key themes to data maturity along a five-stage journey. The seven themes are: Uses, Data, Analysis, Leadership, Culture, Tools and Skills. The five stages are Unaware, Emerging, Learning, Developing and Mastering. An organisation can score at any of the five stages for each of the seven themes. Read our full data maturity framework here.
Figure 1: Data Maturity Themes
Data Maturity Assessment Tool
We have designed a self-assessment tool that organisations can use to evaluate how they score for each of the seven key themes, and therefore their overall data maturity score. The full assessment takes around 20 minutes, there is also a taster version that takes only 5 minutes. This tool is free to use, however we do offer a paid for ‘organisational assessment’ that aggregates scores from multiple people in an organisation to create an overall score.
Theory of Change
Our theory of change outlines how we expect the Data Maturity Assessment tool to have an impact in the short and long term. The theory of change targets specific people and identifies specific problems.
Users
The theory of change for our Data Maturity Assessment tool targets three type of people in the non-for-profit sector:
Leader (CEO, director, trustee)
Data Champion (Data/database manager, analyst or someone responsible for data in impact evaluation, research, fundraising, digital/ICT, marketing, service management or delivery)
Data Strategist (Responsible for data across the whole organisation)
Problems
We expect these people to be experiencing problems with:
Having limited knowledge about data maturity.
Feel like data is not working for their organisation’s cause.
Wanting to get better at data but not knowing where to start.
Struggling to engage leaders and colleagues in discussions about data or driving improvements.
Not knowing what good and great look like, or how well they compare to others. (This could be the sense they are not doing so well and want to check or, after investing effort into data capabilities, feel they are doing better than others and want to check.)
Frustrated by poor and/or outdated systems and tools.
Unable to unlock the value of data for decision making.
Outcomes
By taking a Data Maturity Assessment we predict immediate benefits that bring about tangible actions. These actions lead to results that increase data maturity and provide long-term rewards and benefits.
Immediate benefits
- Advancing knowledge
- Improved understanding about important factors and questions in organisational data maturity.
- Objective reflection of the stage the organisation is currently at.
- Increased awareness of different perceptions about data across the organisation.
- Identified clear priorities to focus on.
- Shared understanding and language about data in the organisation.
- Changing attitudes
- Raised aspirations of what’s possible.
- Increased motivation to improve data maturity.
- Reassured they are on the right track.
Actions
Share results with colleagues and invite them to take the assessment.
Discussed with colleagues where improvement is needed.
Secured leadership support for change and improvement.
Sought external expert advice on data strategy.
Sought funding or resources to implement improvements.
Developed a data strategy/improvement plan.
Results
Secure funding or resources.
Invest in people (existing or new staff roles and responsibilities), tools, advice, and training.
Implement their data strategy/improvement plans.
Long term-rewards & benefits
Improved strategic planning and decision making
Better services and/or products
Greater impact
Increased income
Efficiency savings
Increased credibility and influence
Increased knowledge and expertise
Increased collaboration and data sharing with strategic partners.
Figure 2: Theory of Change for Data Maturity Assessment